{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building an Image Generation Application \n",
    "\n",
    "There's more to LLMs than text generation. It's also possible to generate images from text descriptions. Having images as a modality can be highly useful in a number of areas from MedTech, architecture, tourism, game development and more. In this chapter, we will look into the two most popular image generation models, DALL-E and Midjourney.\n",
    "\n",
    "## Introduction \n",
    "\n",
    "In this lesson, we will cover:\n",
    "\n",
    "- Image generation and why it's useful.\n",
    "- DALL-E and Midjourney, what they are, and how they work.\n",
    "- How you would build an image generation app.\n",
    "\n",
    "## Learning Goals \n",
    "\n",
    "After completing this lesson, you will be able to:\n",
    "\n",
    "- Build an image generation application.\n",
    "- Define boundaries for your application with meta prompts. \n",
    "- Work with DALL-E and Midjourney.\n",
    "\n",
    "## Why build an image generation application?\n",
    "\n",
    "Image generation applications are a great way to explore the capabilities of Generative AI. They can be used for, for example:  \n",
    "\n",
    "- **Image editing and synthesis**. You can generate images for a variety of use cases, such as image editing and image synthesis.  \n",
    "\n",
    "- **Applied to a variety of industries**. They can also be used to generate images for a variety of industries like Medtech, Tourism, Game development and more. \n",
    "\n",
    "## Scenario: Edu4All \n",
    "\n",
    "As part of this lesson, we will continue to work with our startup, Edu4All, in this lesson. The students will create images for their assessments, exactly what images is up to the students, but they could be illustrations for their own fairytale or create a new character for their story or help them visualize their ideas and concepts. \n",
    "\n",
    "Here's what Edu4All's students could generate for example if they're working in class on monuments:\n",
    "\n",
    "![Edu4All startup, class on monuments, Eifel Tower](../images/startup.png?WT.mc_id=academic-105485-koreyst)\n",
    "\n",
    "using a prompt like \n",
    "\n",
    "> \"Dog next to Eiffel Tower in early morning sunlight\"\n",
    "\n",
    "## What is DALL-E and Midjourney? \n",
    "\n",
    "[DALL-E](https://openai.com/dall-e-2?WT.mc_id=academic-105485-koreyst) and [Midjourney](https://www.midjourney.com/?WT.mc_id=academic-105485-koreyst) are two of the most popular image generation models, they allow you to use prompts to generate images.\n",
    "\n",
    "### DALL-E\n",
    "\n",
    "Let's start with DALL-E, which is a Generative AI model that generates images from text descriptions. \n",
    "\n",
    "> [DALL-E is a combination of two models, CLIP and diffused attention](https://towardsdatascience.com/openais-dall-e-and-clip-101-a-brief-introduction-3a4367280d4e?WT.mc_id=academic-105485-koreyst).  \n",
    "\n",
    "- **CLIP**, is a model that generates embeddings, which are numerical representations of data, from images and text.  \n",
    "\n",
    "- **Diffused attention**, is a model that generates images from embeddings. DALL-E is trained on a dataset of images and text and can be used to generate images from text descriptions. For example, DALL-E can be used to generate images of a cat in a hat, or a dog with a mohawk. \n",
    "\n",
    "### Midjourney\n",
    " \n",
    "Midjourney works in a similar way to DALL-E, it generates images from text prompts. Midjourney, can also be used to generate images using prompts like “a cat in a hat”, or a “dog with a mohawk”. \n",
    "\n",
    " \n",
    "\n",
    "![Image generated by Midjourney, mechanical pigeon](https://upload.wikimedia.org/wikipedia/commons/thumb/8/8c/Rupert_Breheny_mechanical_dove_eca144e7-476d-4976-821d-a49c408e4f36.png/440px-Rupert_Breheny_mechanical_dove_eca144e7-476d-4976-821d-a49c408e4f36.png?WT.mc_id=academic-105485-koreyst)\n",
    "\n",
    "*Image cred Wikipedia, image generated by Midjourney*\n",
    "\n",
    "## How does DALL-E and Midjourney Work \n",
    "\n",
    "First, [DALL-E](https://arxiv.org/pdf/2102.12092.pdf?WT.mc_id=academic-105485-koreyst). DALL-E is a Generative AI model based on the transformer architecture with an *autoregressive transformer*.  \n",
    "\n",
    "An *autoregressive transformer* defines how a model generates images from text descriptions, it generates one pixel at a time, and then uses the generated pixels to generate the next pixel. Passing through multiple layers in a neural network, until the image is complete.  \n",
    "\n",
    "With this process, DALL-E, controls attributes, objects, characteristics, and more in the image it generates. However, DALL-E 2 and 3 have more control over the generated image, "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building your first image generation application\n",
    "\n",
    "So what does it take to build an image generation application? You need the following libraries:\n",
    "\n",
    "- **python-dotenv**, you're highly recommended to use this library to keep your secrets in a *.env* file away from the code.\n",
    "- **openai**, this library is what you will use to interact with the OpenAI API.\n",
    "- **pillow**, to work with images in Python.\n",
    "- **requests**, to help you make HTTP requests.\n",
    "\n",
    "\n",
    "1. Create a file *.env* with the following content:\n",
    "\n",
    "    ```text\n",
    "    AZURE_OPENAI_ENDPOINT=<your endpoint>\n",
    "    AZURE_OPENAI_API_KEY=<your key>\n",
    "    ```\n",
    "\n",
    "    Locate this information in Azure Portal for your resource in the \"Keys and Endpoint\" section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "1. Collect the above libraries in a file called *requirements.txt* like so:\n",
    "\n",
    "    ```text\n",
    "    python-dotenv\n",
    "    openai\n",
    "    pillow\n",
    "    requests\n",
    "    ```\n",
    "\n",
    "\n",
    "1. Next, create virtual environment and install the libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "# create virtual env\n",
    "! python3 -m venv venv\n",
    "# activate environment\n",
    "! source venv/bin/activate\n",
    "# install libraries\n",
    "# pip install -r requirements.txt, if using a requirements.txt file \n",
    "! pip install python-dotenv openai pillow requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> [!NOTE]\n",
    "> For Windows, use the following commands to create and activate your virtual environment:\n",
    "\n",
    "    ```bash\n",
    "    python3 -m venv venv\n",
    "    venv\\Scripts\\activate.bat\n",
    "    ````\n",
    "\n",
    "1. Add the following code in file called *app.py*:\n",
    "\n",
    "    ```python\n",
    "    import openai\n",
    "    import os\n",
    "    import requests\n",
    "    from PIL import Image\n",
    "    import dotenv\n",
    "    \n",
    "    # import dotenv\n",
    "    dotenv.load_dotenv()\n",
    "    \n",
    "    # Get endpoint and key from environment variables\n",
    "    openai.api_base = os.environ['AZURE_OPENAI_ENDPOINT']\n",
    "    openai.api_key = os.environ['AZURE_OPENAI_API_KEY']     \n",
    "    \n",
    "    # Assign the API version (DALL-E is currently supported for the 2023-06-01-preview API version only)\n",
    "    openai.api_version = '2023-06-01-preview'\n",
    "    openai.api_type = 'azure'\n",
    "    \n",
    "    \n",
    "    try:\n",
    "        # Create an image by using the image generation API\n",
    "        generation_response = openai.Image.create(\n",
    "            prompt='Bunny on horse, holding a lollipop, on a foggy meadow where it grows daffodils',    # Enter your prompt text here\n",
    "            size='1024x1024',\n",
    "            n=2,\n",
    "            temperature=0,\n",
    "        )\n",
    "        # Set the directory for the stored image\n",
    "        image_dir = os.path.join(os.curdir, 'images')\n",
    "    \n",
    "        # If the directory doesn't exist, create it\n",
    "        if not os.path.isdir(image_dir):\n",
    "            os.mkdir(image_dir)\n",
    "    \n",
    "        # Initialize the image path (note the filetype should be png)\n",
    "        image_path = os.path.join(image_dir, 'generated-image.png')\n",
    "    \n",
    "        # Retrieve the generated image\n",
    "        image_url = generation_response[\"data\"][0][\"url\"]  # extract image URL from response\n",
    "        generated_image = requests.get(image_url).content  # download the image\n",
    "        with open(image_path, \"wb\") as image_file:\n",
    "            image_file.write(generated_image)\n",
    "    \n",
    "        # Display the image in the default image viewer\n",
    "        image = Image.open(image_path)\n",
    "        image.show()\n",
    "    \n",
    "    # catch exceptions\n",
    "    except openai.InvalidRequestError as err:\n",
    "        print(err)\n",
    "\n",
    "    ```\n",
    "\n",
    "Let's explain this code:\n",
    "\n",
    "- First, we import the libraries we need, including the OpenAI library, the dotenv library, the requests library, and the Pillow library.\n",
    "\n",
    "    ```python\n",
    "    import openai\n",
    "    import os\n",
    "    import requests\n",
    "    from PIL import Image\n",
    "    import dotenv\n",
    "    ```\n",
    "\n",
    "- Next, we load the environment variables from the *.env* file.\n",
    "\n",
    "    ```python\n",
    "    # import dotenv\n",
    "    dotenv.load_dotenv()\n",
    "    ```\n",
    "\n",
    "- After that, we set the endpoint, key for the OpenAI API, version and type.\n",
    "\n",
    "    ```python\n",
    "    # Get endpoint and key from environment variables\n",
    "    openai.api_base = os.environ['AZURE_OPENAI_ENDPOINT']\n",
    "    openai.api_key = os.environ['AZURE_OPENAI_API_KEY'] \n",
    "\n",
    "    # add version and type, Azure specific\n",
    "    openai.api_version = '2023-06-01-preview'\n",
    "    openai.api_type = 'azure'\n",
    "    ```\n",
    "\n",
    "- Next, we generate the image:\n",
    "\n",
    "    ```python\n",
    "    # Create an image by using the image generation API\n",
    "    generation_response = openai.Image.create(\n",
    "        prompt='Bunny on horse, holding a lollipop, on a foggy meadow where it grows daffodils',    # Enter your prompt text here\n",
    "        size='1024x1024',\n",
    "        n=2,\n",
    "        temperature=0,\n",
    "    )\n",
    "    ```\n",
    "\n",
    "    The above code responds with a JSON object that contains the URL of the generated image. We can use the URL to download the image and save it to a file.\n",
    "\n",
    "- Lastly, we open the image and use the standard image viewer to display it:\n",
    "\n",
    "    ```python\n",
    "    image = Image.open(image_path)\n",
    "    image.show()\n",
    "    ```\n",
    "    \n",
    "### More details on generating the image\n",
    "\n",
    "Let's look at the code that generates the image in more detail:\n",
    "\n",
    "```python\n",
    "generation_response = openai.Image.create(\n",
    "        prompt='Bunny on horse, holding a lollipop, on a foggy meadow where it grows daffodils',    # Enter your prompt text here\n",
    "        size='1024x1024',\n",
    "        n=2,\n",
    "        temperature=0,\n",
    "    )\n",
    "```\n",
    "\n",
    "- **prompt**, is the text prompt that is used to generate the image. In this case, we're using the prompt \"Bunny on horse, holding a lollipop, on a foggy meadow where it grows daffodils\".\n",
    "- **size**, is the size of the image that is generated. In this case, we're generating an image that is 1024x1024 pixels.\n",
    "- **n**, is the number of images that are generated. In this case, we're generating two images.\n",
    "- **temperature**, is a parameter that controls the randomness of the output of a Generative AI model. The temperature is a value between 0 and 1 where 0 means that the output is deterministic and 1 means that the output is random. The default value is 0.7.\n",
    "\n",
    "There are more things you can do with images that we will cover in the next section.\n",
    "\n",
    "## Additional capabilities of image generation\n",
    "\n",
    "You've seen so far how we were able to generate an image using a few lines in Python. However, there are more things you can do with images.\n",
    "\n",
    "You can also do the following:\n",
    "\n",
    "- **Perform edits**. By providing an existing image a mask and a prompt, you can alter an image. For example, you can add something to a portion of an image. Imagine our bunny image, you can add a hat to the bunny. How you would do that is by providing the image, a mask (identifying the part of the area for the change) and a text prompt to say what should be done.\n",
    "\n",
    "    ```python\n",
    "    response = openai.Image.create_edit(\n",
    "      image=open(\"base_image.png\", \"rb\"),\n",
    "      mask=open(\"mask.png\", \"rb\"),\n",
    "      prompt=\"An image of a rabbit with a hat on its head.\",\n",
    "      n=1,\n",
    "      size=\"1024x1024\"\n",
    "    )\n",
    "    image_url = response['data'][0]['url']\n",
    "    ```\n",
    "\n",
    "    The base image would only contain the rabbit but the final image would have the hat on the rabbit.\n",
    "    \n",
    "- **Create variations**. \n",
    "    Look at our [OpenAI notebook for more information](./oai-assignment.ipynb?WT.mc_id=academic-105485-koreyst)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
